Hierarchical Decision Making

Decision making must be made within an appropriate context; we contend that such context is best represented by a hierarchy of states. The lowest levels of this hierarchy represent the observed raw data, or specific low-level behaviors and decisions. As we ascend the hierarchy, the states become increasingly abstract, representing higher order tactics, strategies, and over-arching mission goals. By representing the hierarchy using probabilistic graphical models, we can readily learn the structure and parameters that define a user’s behavior by observing his activities over time— what data they use, how it is visualized, and what decisions are made. Once learned, the resulting mathematical models may be combined with the techniques of reinforcement learning to predict behavior and anticipate the needs of the user, delivering appropriate data, visualizations, and recommending optimal actions. Keywords—decision making; hierarchical hidden Markov models; reinforcement learning.